This PR introduces support for Low-Rank Adaptation (LoRA) in our LLM inference pipeline, allowing for dynamic model adaptation at inference time. Key changes include:

1. Enhanced LLMGeneratePipeline:

   • Added LoRA weight application functionality
   • Implemented a queue system for handling concurrent requests with different LoRA weights
   • Retained all existing model loading and optimization strategies (8-bit quantization, fp16/bf16 loading, distributed loading)

2. Updated API route:

   • Added support for LoRA weights as an optional parameter
   • Implemented validation for LoRA weight input (base64 encoded string)

3. Maintained compatibility:

   • Preserved all existing pipeline functionality (memory management, device mapping, stable-fast optimization)
   • Ensured backward compatibility for requests without LoRA weights

4. Updated dependencies:

   • Verified and updated requirements.txt to support new functionality

These changes enable users to apply custom LoRA weights to the base model at inference time, allowing for tailored model behavior without changing the underlying model. This feature enhances the flexibility of our inference API while maintaining its performance and efficiency.

Testing:

• Verified functionality with and without LoRA weights
• Tested concurrent requests with different LoRA weights
• Ensured compatibility with existing pipelines and configurations

Open todo's:

• Comprehensive testing with various model sizes and LoRA configurations
• Performance benchmarking to assess impact on inference speed
• Documentation update to explain LoRA usage in API calls